Guidance of invasive medical devices with combined three dimensional ultrasonic imaging system

ABSTRACT

A composite three dimensional ultrasound image containing position or image information of an invasive medical device ( 30 ) is provided. The three dimensional ultrasound image data can be combined with position or image information of the invasive medical device ( 30 ) either prior to or subsequent to volume rendering. the three dimensional ultrasound image data and the interventional system data are oriented and sealed to a common frame of reference and then combined. The volume rendering can be performed either on the ultrasound system ( 12 ), on the interventional system ( 20 ), or separately on both before combining.

This invention relates to ultrasonic diagnostic imaging and, moreparticularly, to the use of three dimensional ultrasonic diagnosticimaging to guide the placement and operation of invasive(interventional) medical devices in the body.

Ultrasonic imaging is commonly used to image the insertion, use oroperation of medical devices and instruments within the body. One suchcommon use of ultrasound imaging is in the conduct of a biopsyprocedure. An ultrasound probe is used to image the pathology ofinterest for the procedure such as a suspected tumor or cyst. The probeis manipulated until the pathology is visible in the image plane. Abiopsy needle attachment for the probe then guides the insertion of thebiopsy needle within the image plane and toward the pathology. Theclinician follows the travel of the needle in the ultrasound image,being careful to keep the probe stationary and the needle within theimage plane until the needle tip reaches the pathology. A specimen isextracted through the needle and the needle is withdrawn from the body.Ultrasonic imaging is thus used to guide the travel of the needle intothe body and to observe the conduct of the biopsy procedure.

Biopsy needles have been designed with their own ultrasonic transmittersor receivers which interact with the imaging probe. Such ultrasonicallyresponsive needles allow the needle and the imaging probe to signal eachother and enable the needle and its tip to be more clearly identified inthe ultrasound image plane. Ultrasonically responsive biopsy needles aredescribed in U.S. Pat. No. 5,158,088, for instance.

The planar imaging techniques are limited in that they provide arestricted, single image view of the internal site of the procedure. Itwould be desirable to provide a greater field of view of the site of theprocedure to enable the clinician or surgeon to better guide and conductthe procedure. Improved imaging would assist biopsy procedures and alsofacilitate a wide range of invasive procedures such as the placement ofstents and cannulae, the dilation or resection of vessels, treatmentsinvolving the heating or freezing of internal tissues, the placement ofradioactive seeds or prosthetic devices such as valves and rings, theguidance of wires or catheters through vessels for the placement ofdevices such as pacemakers, implantable cardiovertors/defibrillators,electrodes, and guide wires, the placement of sutures, staples andchemical/gene sensing electrodes, the guidance or operation of roboticsurgical devices, and the guidance of endoscopic or minimally invasivesurgical procedures. Ultrasonic guidance would thus find expanded use ina broad range of invasive or interventional clinical applicationsincluding cardiac, pulmonary, central and peripheral nervous systemprocedures, gastrointestinal, musculoskeletal, gynecological,obstetrical, urological, ophthalmologic and otorhinolarygologicprocedures.

In accordance with the principles of the present invention, threedimensional ultrasonic imaging is used to guide or monitor the conductof the placement and/or use of invasive (interventional) medical devicessuch as those enumerated above. In one embodiment the location of theinterventional device or its activities are recorded in a threedimensional ultrasound image which consolidates information from boththe three dimensional ultrasonic imaging system and the interventionalsystem. The consolidated image may be viewed on the ultrasound system,on the interventional system, or on the display of a combined ultrasonicimaging and interventional device system. In accordance with a furtheraspect of the present invention the locus of the interventional deviceis ultrasonically scanned in greater detail than the surrounding volumefor greater visual precision and higher frame rate of display of theguidance or use of the interventional device. In accordance with yetanother aspect of the present invention, the results of the invasiveprocedure are recorded in a three dimensional reference system derivedfrom three dimensional ultrasonic image data.

In the drawings:

FIG. 1 illustrates in block diagram form the use of three dimensionalultrasonic imaging to guide or monitor an invasive instrument andprocedure.

FIG. 2 illustrates the three dimensional ultrasonic imaging of acatheter in the heart by a transthoracic transducer probe.

FIG. 3 illustrates in block diagram form the functional subsystems of athree dimensional ultrasonic imaging system suitable for use in anembodiment of the present invention.

FIG. 4 illustrates in block diagram form another embodiment of the useof three dimensional ultrasonic imaging to guide or monitor an invasiveinstrument and procedure.

FIG. 5 illustrates a method for positionally locating an invasivemedical device within the body by means of a two dimensional arraytransducer.

FIG. 6 illustrates a second method for positionally locating an invasivemedical device within the body.

FIG. 7 illustrates the scanning of the volume around an invasive devicewith greater beam density than the surrounding image volume.

FIGS. 8-11 illustrate ultrasound displays of a volume of interesttogether with a greater volumetric field of view containing the volumeof interest.

FIG. 12 illustrates the display of three dimensional, two dimensional,and quantified ultrasonic measures of an interventional site.

FIG. 13 illustrates the display of a detailed three dimensionalultrasonic image of an interventional device along with a greatervolumetric view of the location of the interventional device.

FIG. 14 illustrates the recording of the locus of interventionalprocedures in a three dimensional ultrasonic image.

FIG. 15 illustrates the recording of the loci of interventionalprocedures in a wire frame model derived from three dimensionalultrasonic image data.

FIG. 16 illustrates a simultaneous view of a live three dimensionalimage of an interventional device, a wire frame model recording the lociof interventional procedures, and ECG waveforms relating to the loci.

FIGS. 17-21 are flowcharts illustrating methods for combining imageand/or locational data from a three dimensional ultrasonic imagingsystem and an interventional system.

FIG. 22 illustrates in block diagram form a consolidated system for theconduct of an invasive procedure assisted by three dimensionalultrasonic imaging.

Referring first to FIG. 1, the use of three dimensional ultrasonicimaging to guide or monitor an invasive instrument and procedure isshown in partial block diagram form. On the left side of the drawing isa three dimensional (3D) ultrasonic imaging system including a probe 10having a two dimensional array transducer. The transducer arraytransmits ultrasonic beams over a volumetric field of view 120 undercontrol of an ultrasound acquisition subsystem 12 and receives echoes inresponse to the transmitted beams which are coupled to and processed bythe acquisition subsystem. The echoes received by the elements of thetransducer array are combined into coherent echo signals by theacquisition subsystem and the echo signals along with the coordinatesfrom which they are received (r,θ,φ for a radial transmission pattern)are coupled to a 3D image processor 14. The 3D image processor processesthe echo signals into a three dimensional ultrasonic image which isdisplayed on a display 18. The ultrasound system is controlled by acontrol panel 16 by which the user defines the characteristics of theimaging to be performed.

Also shown in FIG. 1 is an interventional device system. Theinterventional device system includes an invasive (interventional)device 30 which performs a function within the body. In this drawing theinterventional device is shown as a catheter, but it could also be someother tool or instrument such as a needle, a surgical tool such as adissection instrument or stapler or a stent delivery, electrophysiology,or balloon catheter, a therapy device such as a high intensityultrasound probe or a pacemaker or defibrillator lead, a diagnostic ormeasurement device such as an IVUS or optical catheter or sensor, or anyother device which is manipulated and operates within the body. Theinterventional device 30 is manipulated by a guidance subsystem 22 whichmay mechanically assist the maneuvering and placement of theinterventional device within the body. The interventional device 30 isoperated to perform its desired function such as placing an item at adesired location, or measuring, illuminating, heating, freezing, orcutting tissue under the control of an interventional subsystem 20. Theinterventional subsystem 20 also received information from theinterventional device on the procedure being performed, such as opticalor acoustic image information, temperature, electrophysiologic, or othermeasured information, or information signaling the completion of aninvasive operation. Information which is susceptible of processing fordisplay is coupled to a display processor 26. The interventional devicemay also have an active position sensor 32 which is used to provideinformation as to the location of the working tip within the body. Theactive position sensor 32 may operate by transmitting or receivingsignals in the acoustic, optical, radio frequency or electromagneticspectrum and its output is coupled to a device position measurementsubsystem 24. Alternately the sensor 32 may be a passive device such asa diffraction grating which is highly reflective of ultrasonic energytransmitted by the probe 10. Position information of the interventionaldevice is coupled to the display processor 26 when appropriate for theprocessing or display of information concerning the position of theinterventional within the body. Information pertinent to the functioningor operation of the interventional device is displayed on a display 28.The interventional device system is operated by a user through a controlpanel 27.

In the embodiment of FIG. 1 the invasive device 30 is shown as acatheter which is performing a function at the wall of the leftventricle 102 of the heart 100. The full extent of the endocardial wallof the left ventricle is visible by three dimensional ultrasonic imagingof the volumetric field of view 120 of the 3D ultrasonic imaging system.The working tip of the interventional device 30 may include an x-ray,r.f. or ultrasonic device for imaging or ranging the endocardium, or aphysiologic or thermal sensor which conducts diagnostic measurements ofthe endocardium, or an ablation device which treats lesions on theendocardium, or a placement device for an electrode, for example. Thetip of the interventional device is manipulated to a point on the heartwall where such a function is to be performed by operation of theguidance subsystem 22. The interventional device is then commanded toperform its intended procedure by the interventional subsystem 20, andthe location at which the procedure is performed by the device positionmeasurement subsystem 24 which receives or transmits a signal from thesensor 32 at the time of the procedure, for instance.

The invasive procedure may be assisted by monitoring the proceduresimply by visualizing the site of the procedure, the wall of the leftventricle in the foregoing example, by use of the three dimensionalultrasound system. As the interventional device 30 is manipulated withinthe body the three dimensional environment in which the device isoperated can be visualized in three dimensions, enabling the operator toanticipate turns and bends of orifices and vessels in the body and toprecisely place the working tip of the interventional device at thedesired site of the procedure. It is necessary to see a large field ofview in order to provide gross navigation with enough detailedresolution to guide the intervention within the vicinity of the invasivedevice. The operator can maneuver and reposition the probe 10 toconstantly keep the interventional device 30 within the probe'svolumetric field of view. Since in the preferred embodiment the probe 10has a two dimensional array which rapidly transmits and receiveselectronically steered beams, rather than a mechanically swepttransducer, real time three dimensional ultrasonic imaging can beperformed and the interventional device and its procedure observedcontinuously and precisely in three dimensions.

In accordance with a further aspect of this first embodiment of thepresent invention, a signal path 42 connects the ultrasound acquisitionsubsystem 12 of the ultrasound system and the device positionmeasurement subsystem 24 of the interventional device system to allowsynchronization of the imaging system and the interventional device.This synchronization allows image acquisition and interventional deviceoperation to be done at different time interleaved intervals if theoperation of one device would interfere with the other. For example, ifthe interventional device 30 is performing acoustic imaging of the heartor vessel wall, it is desirable for these acoustic intervals to occurwhen they will not be disrupted by acoustic transmissions from the 3Dimaging probe 10. It may be desirable to suspend imaging when theinterventional device is transmitting high energy signals for ablationor some other procedure that would interfere with the imaging signalsfrom the probe 10. The synchronization also enables the ultrasoundsystem to ask for and receive position information from theinterventional device when the ultrasound system is producing a 3Dultrasound image with an enhanced representation of the position of theinterventional device shown in the 3D ultrasound image. The ultrasoundsystem may also ask for and receive position information from theinterventional device when the ultrasound system is recording thelocation of a procedure performed by the interventional device forfuture reference, as will be discussed more fully below.

In accordance with a another aspect of this first embodiment of thepresent invention, image data may be exchanged over a signal path 44between the 3D image processor 14 of the ultrasound system and thedisplay processor 26 of the interventional device system for theformation of a 3D image containing information from both systems. Thisenables the display of an image of the interventional device 30,produced by the interventional device system, as part of a 3D ultrasoundimage produced by the ultrasound system. Such a fusion of the imagingcapabilities of both systems better enables the physician to guide andutilize the interventional device, aided by the extensive threedimensional field of view afforded by the ultrasound system and thedevice image data produced by the interventional device system.

FIG. 2 illustrates practice of the present invention when the threedimensional ultrasound probe used is a transthoracic probe 10. In thisexample the heart 100, shown in partial outline behind the rib cage110,112, is located behind the left side of the rib cage. Outlinedwithin the heart and cross-hatched is the left ventricle 102 of theheart 100. The left ventricle can be accessed for ultrasonic imaging byscanning the heart from between the ribs 110,112 for adult patients and,for some pediatric patients, by scanning upward from below the lowestrib 112. The probe 10 scans the heart from the heart apex 104 asindicated by the outline 120 of the volumetric field of view scanned bythe probe 10. As FIG. 2 illustrates, the left ventricle 102 can be fullyencompasses and scanned by the volumetric field of view directed frombetween the rib cage 110,112.

While this embodiment illustrates phased array scanning of thevolumetric region 120 in a conical field of view, one skilled in the artwill recognize that other scan formats may also be employed such asthose which scan a rectangular or hexagonal pyramidal field of view. Itwill also be appreciated that probes other than transthoracic probes maybe used for three dimensional scanning such as transesophageal probes,intracavity probes such as vaginal or rectal probes, and intervascularprobes such as catheter-mounted transducer probes. While anelectronically scanned two dimensional array transducer is preferred,mechanically scanned arrays may be preferred for some applications suchas abdominal procedures.

FIG. 3 illustrates some of the components of the 3D ultrasound system ofFIG. 1 in further detail. The elements of a two dimensional arraytransducer 50 are coupled to a plurality of microbeamformers 62. Themicrobeamformers control the transmission of ultrasound by the elementsof the array transducer 50 and partially beamform echoes returned togroups of the elements. The microbeamformers 62 are preferablyfabricated in integrated circuit form and located in the housing of theprobe 10 near the array transducer. Microbeamformers, or subarraybeamformers as they are often called, are more fully described in U.S.Pat. Nos. 6,375,617 and 5,997,479. The probe 10 may also include aposition sensor 52 which provides signals indicative of the position ofthe probe 10 to a transducer position detector 54. The sensor 52 may bea magnetic, electromagnetic, radio frequency, infrared, or other type ofsensor such as one which transmits a signal that is detected by avoltage impedance circuit. The transducer position signal 56 produced bythe detector 54 may be used by the ultrasound system or coupled to theinterventional device system when useful for the formation of spatiallycoordinated images containing information from both systems.

The partially beamformed signals produced by the microbeamformers 62 arecoupled to a beamformer 64 where the beamformation process is completed.The resultant coherent echo signals along the beams are processed byfiltering, amplitude detection, Doppler signal detection, and otherprocesses by a signal processor 66. The echo signals are then processedinto image signals in the coordinate system of the probe (r,θ,φ forexample) by an image processor 68. The image signals are converted to adesired image format (x,y,z Cartesian coordinates, for example) by ascan converter 70. The three dimensional image data is coupled to avolume renderer 72 which renders a three dimensional view of thevolumetric region 120 as seen from a selected look direction. Volumerendering is well known in the art and is described in U.S. Pat. No.5,474,073. Volume rendering may also be performed on image data whichhas not been scan converted as described in [U.S. patent applicationSer. No. 10/026,996, filed Dec. 19, 2001 by Alistair Dow and PaulDetmer.] During two dimensional imaging the image plane data bypassesthe volume renderer and is coupled directly to a video processor 76which produces video drive signals compatible with the requirements ofthe display 18. The volume rendered 3D images are also coupled to thevideo processor 76 for display. The system can display individual volumerendered images or a series of volume rendered images showing thedynamic flow and motion of the anatomy being imaged in real time. Inaddition, two volume renderings can be done of a volumetric data setfrom slightly offset look directions, and the two displayedsimultaneously on a stereoscopic display as described in [U.S. patentapplication Ser. No. 60/43,096 (attorney docket 020478), filed Dec. 3,2002 by Jonathan Ziel and entitled “Method and Apparatus to Display 3DRendered Ultrasound Data on an Ultrasound Cart in Stereovision”]. Agraphics processor 74 receives either scan converted image data from thescan converter 70 or unscan-converted image data from the imageprocessor 68 for analysis and the generation of graphics, such as visualemphasis of the tip of an interventional device or the detection of theborder of an organ within the image field. The visual emphasis may beprovided by an enhanced or unique brightness, color, or volume renderingprocess for imaging the tip of the device, for example. The resultantgraphics are coupled to the video processor where they are coordinatedand overlaid with the image for display.

FIG. 4 illustrates another embodiment of the present invention. Thisembodiment differs from that of FIG. 1 in that there is a connection 46from the probe 10 to the device position measurement subsystem 24 of theinterventional device system. In this embodiment the probe 10 includesthe position sensor 52. Rather than process the probe position signal bythe transducer position detector 54 in the ultrasound system, the signalis processed by the same subsystem that processes the position signalfrom the interventional device 30. The two position signals allow adirect correlation of the positions of the interventional device and theprobe to be performed by the interventional device system and used forcoordinated imaging of the two systems.

FIG. 5 illustrates one technique by which the position of theinterventional device 30 is detected by the probe 10. In the drawing atransducer 32 located on the interventional device 30 transmits anacoustic pulse which is received by three elements 51, 51′ and 51″ ofthe transducer array 50. The elements 51 and 51′ are spaced apart fromelement 51″ by known distances a and b. By measuring the times ofarrival of the pulse from the interventional device at the threeelements the position of the transducer 32 with respect to the arraytransducer 50 can be calculated by triangulation. This can be performedby computingx_position=(a ² +v ²(t ₀ ² −t _(a) ²))/2ay_position=(b ² v ²(t ₀ ² −t _(b) ²))/2bz_position=√{square root over (v ² t ₀ ² −x_position² −y_position²)}where t₀ is the time of flight of the pulse to element 51″, t_(a) is thetime of flight to element 51 and t_(b) is the time of flight to element51′ and v is the speed of sound (approximately 1550 m/sec) in the body.

FIG. 6 illustrates a technique for locating the position of theinterventional device in which ultrasound pulses from the imaging probe10 are received by a transducer 32 on the interventional device 30. Thetransducer 32 listens for pulses from the probe 10. The transmit beamwith the strongest signal or the shortest time of flight to thetransducer 32 corresponds to the direction of the transducer 32 withrespect to the transducer array 50. The distance R between the probe 10and the transducer 32 is determined by the time of flight of thetransmitted pulse. In this embodiment the connection 46 or thesynchronizing line 42 would be used to exchange the times oftransmission and/or reception information between the ultrasound systemand the interventional device system.

An interventional device with its own ultrasonic transducer can be usedto provide other locational information. For instance, if theinterventional device has an ultrasonic transducer which is capable oftransmitting and receiving from a distal end of the device, thetransducer can be used for ranging, sending out pulses and receivingechoes from targeted tissues or tissue interfaces and thereby monitoringor measuring or displaying the distance between a distal part of thedevice and nearby anatomy from the time-of-flight of thetransmit-receive interval. In one embodiment of such a device, theoperator can visually observe the device approaching the tissue ofinterest in the three dimensional image, and can also observe a measureof the distance between the device and the tissue. The distance measurecan for example be displayed numerically in centimeters, or in aquantified display such as an M-mode display, in which the progressiveclosure of the device with the tissue can be seen over an interval oftime.

The location of the interventional device may also be detected ifdesired by signal and/or image processing techniques in the ultrasoundsystem. For example, the use of a highly reflective element on theinterventional device 30 such as a diffraction grating as shown in U.S.Pat. No. 4,401,124 may be used to produce a distinctly identifiable echoreturn from the interventional device. Alternatively if theinterventional device has excessive specular reflective characteristics,it may be desirable to use a device which provides better scatteringand/or absorption of ultrasound energy, such as one with a roughened orabsorptive surface. This would allow the system gain to be increased forbetter definition of the tissue structure in the 3D image, while at thesame time the interventional device does not overwhelm the image withbright, strongly reflected echoes. Another alternative is to identifythe shape of the interventional device tip by image processing such asautomated border detection of the tip in the images. Yet anotherapproach is to cause the tip of the interventional device to be vibratedso that it produces a Doppler return as described in U.S. Pat. No.5,095,910. Other embodiments using transducers on the interventionaldevice may be found in U.S. Pat. Nos. 5,158,088 and 5,259,837. Othersensor types for invasive devices such as magnetic field coils may beemployed as described in U.S. Pat. No. 6,332,089.

One of the difficulties when conducting 3D ultrasonic imaging ofinterventional devices is that a wide or deep volumetric field of viewis usually required to adequately image the device in the environs ofits path of travel and procedure. This means that a significant numberof beams must be transmitted and received to adequately scan thevolumetric field of view with the desired resolution and without spatialaliasing. Furthermore, the depth of the field of view requiressignificant times of travel of the transmitted pulses and receivedechoes. These characteristics cannot be avoided as they are mandated bythe law of physics governing the speed of sound in the body. Accordinglya significant amount of time is needed to acquire a full threedimensional image, and the frame rate of display will often be lowerthan desired. FIG. 7 illustrates one solution to this dilemma, which isto use a different scanning methodology in the vicinity of theinterventional device 30 than is employed in the outer reaches of thewide field of view. In the illustrated example, the location of theinterventional device is determined by one of the techniques discussedabove. This information is communicated to the ultrasound acquisitionsubsystem 12, when then transmits a greater beam density in thevolumetric region 122 surrounding the interventional device 30. In theremainder of the volumetric field of view 120 more widely spacedtransmit beams are employed. The space between the widely spacedtransmit beams may be filled in by interpolating synthetic echo signalsif desired. By this technique the volumetric region 122 surrounding theinterventional probe will be shown with higher definition andresolution, enabling the physician to accurately guide and use thedevice at the site of the procedure. The remaining volumetric space willbe shown with less definition but sufficient to orient theinterventional device and procedure in the surrounding tissue. The beamdensity within the volumetric region of the interventional device can beuniformly high with the surrounding space scanned with uniformly lesserbeam density. Alternatively, the highest beam density can be employed inthe vicinity of the device sensor 32, with the beam density decliningwith greater distances from the interventional device. By continuouslytracking the location of the interventional device 30 the volumetricregion 122 is constantly redefined as needed to spatially correspond tothe location of the interventional device 30.

Another variation in beam density which may be employed is to usedifferent orders of received multilines in the proximity of theinterventional device and in the surrounding volume. Variation inmultiline order and transmit beam density can be used together orseparately. For example the same spatial transmit beam density can beused throughout the scanned volume, but higher order multiline (agreater number of differently steered receive lines for each transmitbeam) is used in the vicinity of the interventional device than in thesurrounding volume to produce a more detailed image in the region of theinterventional device. As a second example, a lesser spatial transmitbeam density is used in the surrounding volume than is used in thevicinity of the interventional device. Higher order multiline is thenused in the surrounding volume to fill in the spaces between transmitbeam axes with a relatively large number of received multilines and alower order of multiline reception is used in the vicinity of theinterventional device to fill in the lesser spacing between transmitbeam axes with a relatively fewer number of received multilines for eachtransmit beam. This latter approach will reduce the multiline artifactof echo intensity variation as a function of the distance of eachreceived multiline from the transmit beam axis in the vicinity of theinterventional device where a more detailed view is desired.

FIGS. 8-13 illustrate displays 99 of other techniques for dealing withthis problem of 3D frame rate decline. In FIG. 8 the 3D locus 82 of theinterventional device 30 is scanned with a high beam density and/orframe rate. The volumetric region 82 is shown in the same display 99with a wider planar field of view 80. The plane 80 may be scanned lessfrequently or with a lesser beam density than that of the volumetricregion 82 of the device 30. Different orders of multiline reception mayalso be employed. Scanning the area 80 with a lesser beam density meansthat a greater volume or area can be scanned with the same number oftransmit-receive cycles as were needed to scan the higher beam densityvolume 82. Thus a wider field of view can be scanned with lesser beamdensity and the frame rate of the full image shown in FIG. 8 isincreased. An outline 82′ depicts the location of the interventionaldevice volume 82 in relation to the image plane 80. In FIG. 9 the widerfield of view is provided by a 3D image 84 with a lesser beam densityand/or frame rate than that of the separate interventional device volume82. The location of the interventional device volume 82 in relation tothe greater volume image 84 is indicated by the outline 82′. In FIG. 10the interventional device volume 82 is shown in its correct spatiallocation in relation to the image plane 80. In FIG. 11 theinterventional device volume 82 is shown in its true spatial position inthe more lightly sampled volume 84, the approach depicted in FIG. 7.

FIG. 12 illustrates a display 99 of a three dimensional ultrasonic image400 of a system of vasculature 404,406. An interventional procedure suchas the placement of a stent has been performed at a location 410 that isrecorded in the image. The location 410 was marked by detecting thelocation of the interventional device at the time of the setting of thestent and is thereafter continuously shown in its recorded spatiallocation in the body. The three dimensional image 400 provides acomprehensive view of the locus of the stent, and is produced with alower beam density and/or frame rate than that of a plane 402 of thevolumetric region 400. The image of the plane 402 is shown adjacent tothe volumetric field of view and contains the locational marker 410. Byscanning this image plane 402 with a greater beam density and/or ahigher frame rate the physician is better able to observe the result ofthe interventional procedure. In addition, an image 408 particular tothe procedural site location 410 is shown in the display. This willtypically be a time-based display such as an EKG trace, a spectralDoppler display, an M-mode display, or a color M-mode display, all ofwhich provide physiological information as a function of time.

FIG. 13 illustrates a display 99 employing the techniques shown in FIGS.9 and 11. In this display a wide field of view three dimensionalhexagonal image 300 which is scanned at a lesser beam density and/orframe rate reveals the expanse of a vasculature system 12. Aninterventional device 12 performing a procedure at a point in thevasculature is shown in a more highly resolved volumetric region 306.The interventional device volume 306 is also shown separately in thesame display in a zoomed view 308 in which the point of the procedure isshown in greater detail. This embodiment combines several of thefeatures of the earlier embodiments. Other variations will readily occurto those skilled in the art.

FIG. 14 illustrates an embodiment of the present invention in which theresults of an invasive treatment regime are recorded in a threedimensional ultrasonic image. This drawing illustrates a catheter 30with a working tip which ablates lesions on the heart wall. As eachlesion is treated the location of the treatment is marked by use of theposition sensor 32 or one of the interventional device detectiontechniques discussed above. Another technique for detecting the positionof the catheter is to apply electrodes of different voltages to oppositesides of the body to create a spatial voltage gradient across the body.The sensor 32 detects the impedance at its location in the gradientfield, which corresponds to the spatial location of the catheter workingtip. By using multiple electrodes at different times and places on thebody, the location of the working tip can be sensed in three dimensions.These treatment locations are stored in memory and can be used to mapthe general region where the treatment occurred, such as the heart wall.In accordance with a further aspect of the present invention, thetreatment location information is merged with the three dimensionalultrasonic image data to visually mark the treatment locations on theendocardial wall as shown by the circular markers 92,94 in FIG. 14.Recording the sequence of live three dimensional ultrasound images notonly records the activity of the interventional device but also theprogress and history of the treatment procedure.

FIG. 15 illustrates an embodiment in which the progress and history ofthe treatment procedure is recorded on a model of the anatomy beingtreated, in this case a wire frame model 130. This wire frame model 130is of the heart wall of the left ventricle, and may be formed by borderdetection of the three dimensional data set as described in U.S. Pat.No. 6,491,636 (Chenal et al.) or in U.S. Pat. No. 5,601,084 (Sheehan etal.) or published European patent specification EP 0 961 135 B1 (Mumm etal). As treatment procedures are performed at specific points on theheart wall, those locations are detected by use of the sensor 32 or oneof the interventional device detection techniques discussed above. Thoselocations 132,134,136 are then recorded in the proper spatial locationson the 3D wire frame model 130, where they may appear as unique colorsor intensities. The wire frame model may be a live, real time modelwhich moves in correspondence with the moving heart, or it may be arepresentation of the heart intermediate the heart's shape and size atend systole and end diastole, or it may be a wire frame model of theheart at its moment of greatest expansion at end diastole.

FIG. 16 illustrates a display screen 99 which shows cardiac informationin three ways: a live three dimensional image 100 of the heart and aninterventional device 30, a wire frame model 130 of the chamber of theheart undergoing treatment, and multiple ECG traces 140 taken at pointsof the heart designated by the interventional device 30. The separateECG traces may be labeled, colored, or visually designated in some otherway to show correspondence with the locations at which they wereacquired, which locations may also be shown by markers on the threedimensional ultrasonic image 100, the wire frame model 130, or both.Such a display enables visual monitoring of the live procedure, a visualrecord of the procedures performed, and measurement data acquired at thesites of the procedures.

FIGS. 17-21 are flowcharts which illustrate several ways in which thethree dimensional ultrasonic image data acquired by the ultrasoundsystem can be merged with spatially based data of the interventionaldevice such as location or image data. In the process of FIG. 17acquired 3D ultrasound data is volume rendered by the ultrasound systemto form a 3D ultrasound image in step 202. In step 204 the volumerendered ultrasound image and data identifying the position of the arraytransducer 50 or probe 10 is transmitted to the interventional system.In step 206 the 3D ultrasound image is converted to the frame ofreference of the data of the interventional system. This may involveresealing the coordinate data of the 3D ultrasound image to match thecoordinate scaling of the interventional system data. Once theultrasound image data has been converted, the ultrasound image and theinterventional device data are aligned (step 208) through the use of theprobe and interventional device coordinate information and combined(step 210) to form a consolidated three dimensional image for display 28containing spatially accurate information about the interventionaldevice and/or its procedure.

In the process of FIG. 18 there is no initial volume rendering of the 3Dultrasound image data. Instead, the process begins in step 212 with thescan conversion of the 3D ultrasound image data to form a Cartesianreferenced 3D data set. In step 214 the scan converted 3D data set andthe probe or array transducer position information is transmitted to theinterventional system. In step 216 the 3D ultrasound image data isconverted to the frame of reference of the data of the interventionalsystem. Again, this may be done by resealing the coordinate data of the3D ultrasound image to match the coordinate scaling of theinterventional system data. In step 218 the 3D image data set and theinterventional device data are combined on the basis of their commonreference frame. The merged data sets are then volume rendered by theinterventional system in step 220 to produce a composite threedimensional ultrasonic image containing spatially accurate informationabout the interventional device and/or its procedure.

In the process of FIG. 19 the data acquired by the interventionaldevice, such as position or image data, is transmitted to the ultrasoundsystem in step 230. In step 232 the interventional device data isconverted to the frame of reference of the ultrasound probe ortransducer as by resealing the data In step 234 the interventionaldevice data is combined with 3D scan converted ultrasonic image data. Instep 236 the combined data is volume rendered to form a threedimensional ultrasonic image containing spatially relevant informationfrom the interventional system. The images thus produced are displayedon the ultrasound system display 18.

In the process of FIG. 20 volume rendered video data produced by theinterventional system is transmitted to the ultrasound system in step240. In step 242, three dimensional ultrasound image data is rescaledand oriented to match the frame of reference of the interventionalsystem data. The three dimensional ultrasound image data is thenrendered in step 244 from the frame of reference or perspective that wasused in the rendering of the interventional system data. Theinterventional system video data and the 3D ultrasound image data, nowrendered to the same frame of reference, may now be combined to form aconsolidated three dimensional image in step 246.

In the process of FIG. 21 volume rendered 3D ultrasound video data istransmitted to the interventional system in step 250. In step 252interventional system image data is resealed and oriented to the frameof reference of the 3D ultrasound rendering. In step 254 theinterventional system image data is rendered to the same frame ofreference as that of the 3D ultrasound image. In step 256 the commonlyreferenced interventional system video data and the 3D ultrasound videodata are combined into a consolidated image.

It will be apparent to those skilled in the art that three types ofcoordinate transformations between the three dimensional ultrasound dataand the interventional device location data are possible. Theinterventional device data can be transformed to the coordinate systemof the 3D ultrasound image, or the 3D ultrasound image data can betransformed to the coordinate system of the interventional device data,or both sets of data can be translated to a user-chosen scale or frameof reference.

The principles of the foregoing data combining processes may be appliedin the embodiment of FIG. 22 which is a combined 3D ultrasonic imagingand interventional device system. In this embodiment the interventionaldevice 30 is manipulated by the guidance system 22 and operated toperform its procedure under control of the intervention subsystem 20. Attimes signaled by the intervention subsystem or the ultrasoundacquisition subsystem the device position measurement subsystem 24acquires locational information of the imaging probe 10, theinterventional device 30, or both. The 3D ultrasonic image data acquiredby the probe 10 and the ultrasound acquisition subsystem 12, and theinterventional data acquired by the intervention subsystem, and thelocation data of the probe 10, the interventional device 30, or both,are then processed to form a consolidated 3D ultrasound image by the 3Dimage processor 14. The 3D image containing interventional device and/orprocedure information is then displayed on the display 18. The entireimaging and interventional system is controlled by a user control panel29.

As used herein the terms “surgical region”, “surgical guidance”, and“surgery” encompass any medical procedure in which a medical instrumentor device is introduced into the body for a medical purpose.

1. A method of producing a three dimensional ultrasonic image containingspatial information of the placement or operation of an invasive medicaldevice comprising: acquiring a three dimensional ultrasonic image dataset from a volumetric region containing an invasive medical device;volume rendering the three dimensional ultrasonic image data set toproduce a three dimensional ultrasonic image; transmitting the threedimensional ultrasonic image to an interventional system; converting thethree dimensional ultrasonic image to a frame of reference of theinterventional system; aligning the three dimensional ultrasonic imagewith position or image data of the invasive medical device; andcombining the position or image data of the invasive medical device withthe three dimensional ultrasonic image.
 2. The method of claim 1,wherein acquiring further comprises acquiring a three dimensionalultrasonic image data set with an array transducer; and furthercomprising acquiring transducer position information, whereintransmitting further comprises transmitting the transducer positioninformation to the interventional system.
 3. The method of claim 1,wherein the method is performed to produce three dimensional ultrasonicimages for real time display.
 4. A method of producing a threedimensional ultrasonic image containing spatial information of theplacement or operation of an invasive medical device comprising:acquiring a three dimensional ultrasonic image data set from avolumetric region containing an invasive medical device; scan convertingthe three dimensional ultrasonic image data set; transmitting the scanconverted three dimensional ultrasonic image data set to aninterventional system; converting the three dimensional ultrasonic imagedata set to a frame of reference of the interventional system; combiningthe three dimensional ultrasonic image data set with position or imagedata of the invasive medical device; and volume rendering the combineddata to produce a composite three dimensional image.
 5. The method ofclaim 4, wherein acquiring further comprises acquiring a threedimensional ultrasonic image data set with an array transducer; andfurther comprising acquiring transducer position information, whereintransmitting further comprises transmitting the transducer positioninformation to the interventional system.
 6. The method of claim 4,wherein the method is performed to produce composite three dimensionalimages for real time display.
 7. A method of producing a threedimensional ultrasonic image containing spatial information of theplacement or operation of an invasive medical device comprising:transmitting position or image data of an invasive medical device to anultrasonic imaging system; converting the device position or image datato a frame of reference of the ultrasonic imaging system; acquiring athree dimensional ultrasonic image data set from a volumetric regioncontaining the invasive medical device; scan converting the threedimensional ultrasonic image data set; combining the scan convertedthree dimensional ultrasonic image data set with the position or imagedata of the invasive medical device; and volume rendering the combineddata to produce a composite three dimensional image.
 8. The method ofclaim 7, wherein the method is performed to produce composite threedimensional images for real time display.
 9. A method of producing athree dimensional ultrasonic image containing spatial information of theplacement or operation of an invasive medical device comprising:transmitting volume rendered video data from an interventional system toan ultrasonic imaging system; acquiring a three dimensional ultrasonicimage data set from a volumetric region containing an invasive medicaldevice; scaling and orienting the three dimensional ultrasonic imagedata set to a frame of reference of the interventional system; volumerendering the three dimensional ultrasonic image data set to produce athree dimensional ultrasonic image; and combining the interventionalsystem video data and the three dimensional ultrasonic image.
 10. Themethod of claim 9, wherein the method is performed to produce combinedinterventional system video data and three dimensional ultrasonic imagesfor real time display.
 11. A method of producing a three dimensionalultrasonic image containing spatial information of the placement oroperation of an invasive medical device comprising: transmitting volumerendered three dimensional ultrasound video data of a volumetric regioncontaining an invasive medical device to an interventional systems;scaling and orienting interventional system video data to a frame ofreference of the ultrasound video data; volume rendering theinterventional system video data; and combining the volume renderedinterventional system video data and the volume rendered threedimensional ultrasound video data.
 12. The method of claim 11, whereinthe method is performed to produce combined volume renderedinterventional system video data and three dimensional ultrasound videodata for real time display.